Efficient permutation testing of variable importance measures by the example of random forests

Hypothesis testing of variable importance measures (VIMPs) is still the subject ongoing research. This particularly applies to random forests (RF), for which VIMPs are a popular feature. Among recent developments, heuristic approaches parametric have been proposed whose distributional assumptions based on empirical evidence. Other formal tests under regularity conditions were derived analytically. But these can be computationally expensive or even practically infeasible. problem also occurs with non-parametric permutation tests, are, however, distribution-free and generically applied any kind prediction model VIMP. Embracing this advantage, it use sequential p-value estimation reduce computational costs associated conventional tests. These high in case complex models. Therefore, RF's widely used VIMP (pVIMP) serves as practical relevant application example. The results simulation studies confirm theoretical properties that is, type-I error probability controlled at nominal level power maintained considerably fewer permutations needed compared testing. numerical stability methods investigated two additional studies. In summary, theoretically sound possible greatly reduced costs. Recommendations given. A respective implementation pVIMP provided through accompanying R package rfvimptest.